Detection of objects in digital images

ABSTRACT

A system and method to detect objects in a digital image. At least one image representing at least one frame of a video sequence is received. A given color channel of the image is extracted. At least one blob that stands out from a background of the given color channel is identified. One or more features are extracted from the blob. The one or more features are provided to a plurality of pre-learned object models each including a set of pre-defined features associated with a pre-defined blob type. The one or more features are compared to the set of pre-defined features. The blob is determined to be of a type that substantially matches a pre-defined blob type associated with one of the pre-learned object models. At least a location of an object is visually indicated within the image that corresponds to the blob.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims priority to U.S. ProvisionalPatent Application Ser. No. 61/323,673 filed Apr. 13, 2010 thedisclosure of which is hereby incorporated by reference in its entirety.

The present patent application is related to commonly owned U.S. patentapplication Ser. No. ______, Attorney Docket No. YOR920100215US2,entitled “Object Recognition Using Haar Features and Histograms ofOriented Gradients”, filed on ______, the entire teachings of whichbeing hereby incorporated by reference.

FIELD OF THE INVENTION

The present invention generally relates to the detection of objects indigital images, and more particularly relates to detecting objects indigital images using camera sensors deployed in a human assistiveenvironment.

BACKGROUND OF THE INVENTION

Digital image based object detection, especially with respect to trafficsign recognition (TSR), has seen increased attention over the past fewyears. For example, object detection systems are currently beingimplemented in advanced driver assistance systems (ADAS). Conventionalobject detection methods usually involve two stages. First, in thedetection stage, image regions that contain candidates of target objectsare detected or localized. Then, in the recognition stage, such regionsare further analyzed to recognize the specific content. However, theseconventional object detection systems and methods generally require alarge amount of computing resources, have slow detection speeds, and canbe inaccurate at times.

SUMMARY OF THE INVENTION

In one embodiment, a method for detecting objects in a digital image.The method comprises receiving at least one image representing at leastone frame of a video sequence of an external environment. A given colorchannel of the at least one image is extracted. At least one blob thatstands out from a background of the given color channel that has beenextracted is identified. One or more features are extracted from the atleast one blob that has been identified. The one or more features thathave been extracted are provided to a plurality of pre-learned objectmodels each comprising a set of pre-defined features associated with apre-defined blob type. The one or more features are compared to the setof pre-defined features of each pre-learned object model in response tothe providing. The at least one blob is determined, in response to thecomparing, to be of a type that substantially matches a pre-defined blobtype associated with one of the pre-learned object models. At least alocation of an object is visually indicated within the at least oneimage that corresponds to the at least one blob in response to thedetermining.

In another embodiment, an information processing system for detectingobjects in a digital image is disclosed. The information processingsystem comprises a memory and a processor that is communicativelycoupled to the memory. The information processing system also comprisesan object detection system that is communicatively coupled to the memoryand the processor. The object detection system is configured to performa method. The method comprises receiving at least one image representingat least one frame of a video sequence of an external environment. Agiven color channel of the at least one image is extracted. At least oneblob that stands out from a background of the given color channel thathas been extracted is identified. One or more features are extractedfrom the at least one blob that has been identified. The one or morefeatures that have been extracted are provided to a plurality ofpre-learned object models each comprising a set of pre-defined featuresassociated with a pre-defined blob type. The one or more features arecompared to the set of pre-defined features of each pre-learned objectmodel in response to the providing. The at least one blob is determined,in response to the comparing, to be of a type that substantially matchesa pre-defined blob type associated with one of the pre-learned objectmodels. At least a location of an object is visually indicated withinthe at least one image that corresponds to the at least one blob inresponse to the determining.

In yet another embodiment, a computer program product for detectingobjects in a digital image is disclosed. The computer program productcomprises a storage medium readable by a processing circuit and storinginstructions for execution by the processing circuit for performing amethod. The method comprises receiving at least one image representingat least one frame of a video sequence of an external environment. Agiven color channel of the at least one image is extracted. At least oneblob that stands out from a background of the given color channel thathas been extracted is identified. One or more features are extractedfrom the at least one blob that has been identified. The one or morefeatures that have been extracted are provided to a plurality ofpre-learned object models each comprising a set of pre-defined featuresassociated with a pre-defined blob type. The one or more features arecompared to the set of pre-defined features of each pre-learned objectmodel in response to the providing. The at least one blob is determined,in response to the comparing, to be of a type that substantially matchesa pre-defined blob type associated with one of the pre-learned objectmodels. At least a location of an object is visually indicated withinthe at least one image that corresponds to the at least one blob inresponse to the determining.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying figures where like reference numerals refer toidentical or functionally similar elements throughout the separateviews, and which together with the detailed description below areincorporated in and form part of the specification, serve to furtherillustrate various embodiments and to explain various principles andadvantages all in accordance with the present invention, in which:

FIG. 1 is a block diagram illustrating a high level overview of a systemfor detecting objects in digital images according to one embodiment ofthe present invention;

FIG. 2 is a block diagram illustrating a feature-based detector used bythe system of FIG. 1 according to one example of the present invention;

FIG. 3 shows one example of a frame/image comprising two circulartraffic signs to be detected by the feature-based detector of FIG. 2according to one example of the present invention;

FIG. 4 shows one example of an edge map for the frame/image shown inFIG. 3 according to one example of the present invention;

FIG. 5 shows one example of a voting image for the edge map of FIG. 4according to one example of the present invention;

FIG. 6 shows one example of identifying a search area within aframe/image according to one example of the present invention;

FIG. 7 shows one example of tracking objects between frames anddetecting emerging objects in a frame according to one example of thepresent invention;

FIG. 8 illustrating one example of partial regions of an image/frame fordetecting emerging objects according to one example of the presentinvention;

FIG. 9 shows one example of an SVM-based detector used by the system ofFIG. 1 according to one example of the present invention;

FIG. 10 shows one example of a frame/image comprising a traffic sign tobe detected by the SVM-based detector of FIG. 9 according to one exampleof the present invention;

FIG. 11 shows one example of a blob that has been detected in theframe/image of FIG. 10 according to one example of the presentinvention;

FIG. 12 shows that the orientation of the blob in FIG. 11 has beenadjusted according to one example of the present invention;

FIG. 13 is an operational flow chart illustrating SVM-based detectortraining and classification according to one embodiment of the presentinvention;

FIG. 14 shows one example of SVM-based feature extracting according toone embodiment of the present invention;

FIG. 15 shows various examples of regions within a bounding box used todetermine the shape of a detected blob according to one embodiment ofthe present invention;

FIG. 16 shows various examples of Haar features;

FIG. 17 shows various examples of Histogram of Oriented Gradientsfeatures;

FIG. 18 shows one example of a cascade-boosted detector used by thesystem of FIG. 1 according to one embodiment of the present invention;

FIG. 19 shows one example of a frame/image comprising a circular trafficsign detected by the cascade-boosted detector of FIG. 18 according toone embodiment of the present invention;

FIG. 20 is an operational flow diagram illustrating one example of aprocess for detecting objects in a digital image using thecascade-boosted detector of FIG. 18 according to one embodiment of thepresent invention;

FIG. 21 illustrates one example of a cascaded classifier according toone embodiment of the present invention;

FIG. 22 shows a table created after training classifiers based on Haarand HOG features according to one embodiment of the present invention;

FIG. 23 is an operational flow diagram illustrating one process fortraining the cascade-boosted detector of FIG. 18 according to oneembodiment of the present invention;

FIG. 24 is an operational flow diagram illustrating one process fordetecting objects in a digital image using the SVM-based detector ofFIG. 9 according to one embodiment of the present invention; and

FIG. 25 is a block diagram illustrating a more detailed view of aninformation processing system according to one embodiment of the presentinvention.

DETAILED DESCRIPTION

As required, detailed embodiments of the present invention are disclosedherein; however, it is to be understood that the disclosed embodimentsare merely examples of the invention, which can be embodied in variousforms. Therefore, specific structural and functional details disclosedherein are not to be interpreted as limiting, but merely as a basis forthe claims and as a representative basis for teaching one skilled in theart to variously employ the present invention in virtually anyappropriately detailed structure and function. Further, the terms andphrases used herein are not intended to be limiting; but rather, toprovide an understandable description of the invention.

The terms “a” or “an”, as used herein, are defined as one or more thanone. The term plurality, as used herein, is defined as two or more thantwo. The term another, as used herein, is defined as at least a secondor more. The terms including and/or having, as used herein, are definedas comprising (i.e., open language). The term coupled, as used herein,is defined as connected, although not necessarily directly, and notnecessarily mechanically.

Operating Environment

According to one embodiment, FIG. 1 illustrates a general overview ofone operating environment 100. In particular, FIG. 1 shows aninformation processing system 102 that can be implemented within avehicle such as an automobile, motorcycle, and the like. Additionally,the system 102 can be communicatively coupled to a user assistedtraining environment for training purposes. The system 102 includes,among other things, an object detection system (ODS) 104, an imagingsystem 106, and an image database 108 comprising one or more images 110.The images 110, in one embodiment, represent frames of a video sequenceof an external environment. The ODS 104, in one embodiment, comprises afeature-based detector 112 and/or one or more machine-learning-baseddetectors 114, 116. It should be noted that, in some embodiments, one ormore of the above components reside outside of and are coupled to theinformation processing system 102.

The ODS 104, in one embodiment, operates in real time to automaticallydetect various objects of a given desired object type/class fromframes/images 110 captured by the imaging system(s) 106. Throughout thefollowing discussion, signs, such as traffic signs, are used as theobject of interest (target objects) for the ODS 104. However, this isonly one example of an object that is detectable by the ODS 104. Otherexamples of objects that are detectable by the ODS 104 include, but arenot limited to, vehicles, headlights, pedestrians, animals, and/or thelike. Sign detection is important, especially in advanced driverassistance systems (ADAS). For example, traffic signs are designed tonot only regulate the traffic, but also indicate the state of the road.This helps guide and warn drivers and pedestrians when traveling on theroad associated with the traffic signs. As will be shown in greaterdetail below, the ODS 104 can utilize a feature-based detector and/or alearning based detector that is able to detect or localize various typesof traffic signs from real-time videos or still images.

Examples of traffic signs applicable to one or more embodiments of thepresent invention can include circular signs having a border of acertain color such as (but not limited to) red, triangular signs havinga border of a certain color, and inverted triangular signs having aborder of a certain color. It should be noted that although thefollowing discussion utilizes traffic signs shown as the objects ofinterest for the ODS 104, one or more embodiments of the presentinvention are not limited to these specific traffic signs (or trafficsigns in general).

Feature Based Detection

FIG. 2 shows a more detailed view of the ODS 104. In the example shownin FIG. 2, the ODS 104 comprises a feature-based detector 112. Thefeature-based detector 112 comprises an edge mapper 202, a voting imagegenerator 204, a color/edge processor 206, an analyzer 208, and anobject detector 210. The feature-based detector 112 utilizes thesecomponents to detect objects of interest within frames/images 110. Inthe following examples, feature-based detector 112 utilizes thesecomponents to detect signs based on their symmetric nature. A moredetailed discussion of radial symmetry can be found in Loy, G.,Zelinsky, A.: Fast radial symmetry for detecting points of interest:IEEE Transactions on Pattern Analysis and Machine Intelligence 25 (2003)959-973, which is hereby incorporated by reference in its entirety.

One type of sign that the feature-based detector 112 can detect arecircular signs. Circles are radially symmetric, therefore, for a givenpixel p and its gradient {right arrow over (g)}, which is calculatedusing an edge operator that yields orientation, if p lays on the arc ofa circle, then the center of the circle (denoted by c) would be in thedirection of gradient {right arrow over (g)} and at the distance of itsradius (denoted by r). In one example, the gradient {right arrow over(g)}(p) can point away from the circle center or towards the circlecenter. See, for example, Barnes, N., Zelinsky, A.: Real-time radialsymmetry for speed sign detection: IEEE Intelligent Vehicles Symposium(2004), which is hereby incorporated by reference in its entirety.

Therefore, for a given frame/image 310 (FIG. 3) in the image database108, the edge mapper 202 of the feature-based detector 112 generates anedge map 402 (FIG. 4) that identifies edges within the image 310. As canbe seen in FIG. 3, the image 310 comprises two circular signs 302, 304.The edge mapper 202 can utilize various techniques to identify sharpdiscontinuities in the intensity profile of the image 310. The edgemapper 202 uses one or more edge detectors, which are operators thatcompute differences between pairs of neighboring pixels, to generate theedge map 402. High responses to these operators are identified as edgepixels. The edge map 402 can be computed in a single scan through theimage 310. Examples of edge detectors are the Gradient- andLaplacian-type edge finders and edge operators such as Sobel. The edgemap 402 of FIG. 4 was generated using the Sobel operator on the image310 of FIG. 3.

The analyzer 208 then analyzes the edge map 402 to estimate, for everyedge pixel p in the edge map 402, the center of the circle that ppossibly belongs to. One method for estimating this center is by castinga vote to the pixel p at a radius r along or against the direction ofits gradient. The more votes that a pixel gets from other pixels, themore likely it is a true circle center. The voting image generator 204then generates a voting image 502 (FIG. 5) based on the above estimationprocess. As can be seen from FIG. 5, pixels 504, 506 that surround thecenter of the two circular signs 302, 304 are much brighter than theothers. The object detector 210 then analyzes the voting image 502 andidentifies these brighter pixels 504, 506 and identifies the circularentities 302, 304 associated with these pixels as traffic signs.

Another type of sign that the feature-based detector 112 can detect aretriangular signs. The feature-based detector 112 performs a processsimilar to that discussed above with respect to circular signs withadditional knowledge about the pattern of the edge orientations. Forexample, assuming that a triangle inscribes upon a circle with rdenoting the perpendicular distance from the centroid to the edge, thefeature-based detector 112 performs sign detection based on thefollowing two cues: 1) for each edge pixel p that lies on a triangle,its gradient points to a straight line that goes through the center andis at the distance of r. In other words, rather than gradient elementsvoting for a single point, a line of votes is cast describing possibleshape centroid positions that account for the observed gradient element;and 2) triangular sign is equi-angular, which prompts the feature-baseddetector 112 to apply a rotationally invariant measure to check how wella set of edges fit a particular angular spacing. See, for example, Loy,G., Barnes, N.: Fast shape-based road sign detection for a driverassistance system: Proceedings of IEEE/RSJ International Conference onIntelligent Robots and Systems (2004) 70-75, which is herebyincorporated by reference in its entirety. The image pixels in theneighborhood of the triangle centroid stand out in the input imagesimilar to the pixels 504, 506 in FIG. 5 for a circular sign.

The feature-based detector 112 also applies post-processing to reducefalse positives by exploiting both color and edge information. Forexample, the color and edge processor 206 of the feature-based detector112 checks if the gravity center of all color pixels of interest (e.g.,red) within the bounding box of the candidate sign is close enough toits detected centroid. This ensures that all color pixels of interest,if not all on the edges, be symmetrically distributed within thebounding box. Moreover, the number of color pixels of interest should bewithin certain percentage range of the sign's size. Also, the color andedge processor 206 validates the triangle geometry by checking theangles formed by three edges. Specifically, the color and edge processor206 performs color segmentation on the candidate image region, retainingonly color pixels of interest. The color and edge processor 406 thensplits the region into maps that contain pixels with 0°, 60° and 120°orientation, respectively.

Next, the color and edge processor 206 performs an interactive methodsuch as, but not limited to, the Random Sample Consensus (RANSAC)procedure to fit a line on pixels in each map with the best effort. See,for example, Fischler, M., Bolles, R.: Random sample consensus: Aparadigm for model fitting with applications to image analysis andautomated cartography: Communication of the ACM 24 (1981) 381-395, whichis hereby incorporated by reference in its entirety. The color and edgeprocessor 406 then verifies the angles between these three lines, aswell as checking the size of formed triangle.

For example, there are typically some pixels with the same direction ofgradient in the corner opposite the line in each sub image. These cornerpixels throw off a least squares regression. Therefore, RANSAC isapplied to bypass them. The RANSAC procedure for 2D lines can beoutlined as follows. First, two points are randomly selected and theline they define is calculated. Then the number of points in the imagewithin some tolerance of that line is counted. If the number of pointsaccounted for is above a given threshold the process continues.Otherwise, the first and second steps are repeated until the maximumiteration limit is reached or a suitable line is found. Then, a leastsquares process is used to find a new line with the points selected inthe third step as input. One embodiment improves upon the aboveprocedure by immediately rejecting lines calculated in the first stepthat have a slope outside of a given tolerance around the expected valuefor a 0°/60°/120° line. This prevents the algorithm from selectingcandidate lines perpendicular to the real line, passing through theopposite corner. It should be noted that the above discussion is alsoapplicable to other geometric shapes as well and not just circular andtriangular shapes.

Once an object such as a traffic sign (e.g., circular or triangular), isdetected by the object detector 410, the feature-based detector 112 cantrack it across the frames (e.g., across sequential images) so as toimprove the processing speed. Nevertheless, to avoid tracking falselydetected objects, the feature-based detector 112 can implement adetect-by-tracking strategy. For example, the feature-based detector 112can first track the search area for a previously-detected traffic sign,and then confirm its exact location by performing the detection withinthe search area.

In more detail, objects are tracked based on the tracking informationobtained from the previous frame, so that the ODS 104 can quickly adaptto the changing driving situation (e.g. changing driving direction,changing speed, etc.). For example, assume that an object A has beendetected at frame f. Tracking this object to frame f+1 involves thefollowing two sub-tasks: 1) identifying a search area S in frame f+1 forlocalizing object A and 2) actually localizing object A within area S.

To accomplish the first sub-task, the size of the search area S isdetermined based on the size and moving direction of object A detectedin the previous frame. For example, if A is detected in frame f 602(FIG. 6) for the first time, then due to the lack of moving information,S is set to be a larger area centering on object A in frame f+1 604. InFIG. 6, the area 606 denoted by the diagonal pattern denotes the searcharea S for object A.

Otherwise, if object A was also previously detected in frame f−1, thenA's moving information (i.e. displacement in both x and y directions)from the frame pair f−1 and f is obtained. This moving information isused to derive the possible position for object A in frame f+1. Forexample, as shown in FIG. 7, the center of area S 702 in frame f+1 708is calculated from the center of A in frame f 706 based on object A'smoving information derived from frame f−1 704 to f 706. As for the sizeof S 702, it is derived based on the following two factors: 1) the sizechange of object A from frame f−1 704 to f 706; and 2) the size ofobject A in frame f 706. If object A has a large size, meaning that itis close to the host car, then it tends to change more in terms of itssize for the subsequent frames. In contrast, if object A is small,meaning that it is far away from the host car, then object A's sizechange would be small or even unnoticeable for the subsequent fewframes.

To fulfill the second sub-task, i.e., to actually localize the objectwithin search area S, the following is performed. Once the search area S702 is detected, the same object detection process is carried out, asdiscussed above, to locate the object within S 702. Compared toperforming the detection over the entire image, by constraining thedetection within a small search area, the performance of the processincreases.

In addition to the above, one or more embodiments perform a partial scanfor detecting emerging objects. Using frame f+1 708 shown in FIG. 7 asan example, if target objects are only searched for within area S 702,object B that just appears in frame f+1 708 (or when object B 710becomes large enough to be detected) one or more embodiments are able todetect this emerging object. For example, one embodiment searches therest of area in the frame. However, the benefit of restricting thesearch within a relatively small area is then lost.

In another embodiment, a pre-defined portion 804, 806, 808, 810 withineach frame 802 is searched, in addition to the area S 702, as shown inFIG. 8. For example, a frame 802 is divided into four quadrants 804,806, 808, 810, with each quadrant b sequentially searched in every fourframes. Therefore, at any frame, only ¼ of its size is being searchedfor, plus the identified search area S 702 if an object is previouslydetected. In this case, any emerging object will be eventually detectedand then delayed at most by 3 frames. In one embodiment, consideringthat an object may lay on the boundary of two quadrants, some overlap isallowed, which is set to be the minimum radius r for searching thetarget traffic signs, between every two quadrants. To further speed upwithout sacrificing the performance, the range of object size to besearched in this case could be set to small.

Machine Learning Based Detection

As discussed above, the ODS 104 can also include one or moremachine-learning-based detectors 114, 116. For example, in oneembodiment, the ODS 104 comprises a Support Vector Machine (SVM)-baseddetector 114. FIG. 9 shows one example of the SVM-based detector 114. Inparticular, FIG. 9 shows that the SVM-based detector 114 comprises amulti-level detection framework. Specifically, the first level includesan extractor module 902 that receives an incoming frame/image 1010 andextracts a binarized color channel of the frame/image 1010. FIG. 10shows one example of a frame/image 1010 that comprises a rectangularsign 1002. The second level includes a blob detection module 904, whichperforms connected component analysis on the extracted binarized colorchannel to obtain a list of blobs. FIG. 11 shows the binarized colorchannel 1102 of the image 1010 in FIG. 10 where the detected blob 1104is bounded by a white rectangle 1106.

The third level includes one or more binary classifiers or a multi-classSVM (Support Vector Machine) classifier 906 each applying an SVMlearning approach to recognize different image blobs, which includecandidates of circles 908, triangles 910, inverted triangles 912, andoptional other objects 914. Note that the optional other object category914 mainly includes those elements such as other geometric shapes knownto not be associated with the objects types not being monitored for. Inan embodiment that utilizes one or more binary classifiers, there is apre-trained classifier for each object type to be detected. For example,in one embodiment there is a pre-trained classifier that recognizescircles, a pre-trained classifier that recognizes triangles, and apre-trained classifier that recognizes inverted triangles. Thesepre-trained classifiers can also be referred to pre-trained objectmodels each comprising a set of pre-defined features associated with apre-defined blob type. It should be noted that the following discussionwith respect to a multi-class classifier also applies to the separatepre-trained classifiers Finally, in the fourth level a frame-leveldetection process 916 is carried out to determine if a detected blob forthe current frame/image is an object of interest, such as a sign, basedon at least the SVM classification result.

The following is a more detailed discussion on the extractor module 902of the SVM-based detector 114. In the current example, the SVM-baseddetector 114 is targeting traffic signs that all have enclosed bordersof a given color, such as red. As a result, these signs can be treatedas blobs in the red channel of the image, and be classified accordingly.Therefore, the extractor module 902 receives an incoming frame I, suchas the frame/image 1010 shown in FIG. 10, and extracts its binarized redchannel I^(R). Specifically, the extractor module 2002 first converts Ifrom the original Red, Green, Blue (RGB) space into Hue, Saturation,Insensity (HIS) space. Then, for each pixel that appears to be red, theextractor module 902 sets its values, for example, to 255 in I^(R);otherwise, 0. Here, a color is defined to be red if its hue is between 0and 30, or 330 and 360, and its saturation is larger than 45. However,other values can be applied as well.

With respect to the blob detection module 904 of the SVM-based detector114, this module 904 applies a standard connected component analysis toidentify spots in an image 1010 of a given color, such as red in thisexample. For example, the blob detection module 904 performs a connectedcomponent analysis on I^(R) to obtain a list of blobs. These blobs arethen passed onto the SVM classifier layer 906 for type recognition. Notethat by applying such spatial attention function at this level of theODS 104, a substantial computational effort can be saved from runningthe classification engine at all possible positions, scales, ororientations within the image. The frame-level object detection layer916 then detects objects of interest based on the blob classification inthe SVM classifier layer 906 and presents detected objects of interestto a user accordingly. For example, FIG. 11 shows one example ofvisually indicating a location of a detected object of interest, such asthe blob 1104 representing the triangular sign 1102 in frame/image 1010of FIG. 10, by surrounding the blob 1104 with a bounding box 1106.

The following is a more detailed discussion on the multi-class SVMclassifier 906. Two phases are involved in this module 906: SVM trainingand SVM classification. In one embodiment, the training phase produces amulti-class SVM model, which has learned the following three differenttypes of blobs: circle 908, triangle 910, and inverted triangle 912.Optionally, a fourth type of blob, other 914 (anything other than atarget object), can be learned as well. The classification phase thenuses such model to recognize the type for an unknown blob.

FIG. 13 illustrates a flowchart for the above two processes.Specifically, during the training phase, a set of images 110, at step1302, is identified that include representative data in terms of targetobjects. Qualified blobs, at step 1304, are then detected within eachframe and matched with the ground truth (e.g., human annotations of thetarget objects in frames/images) in terms of their spatial locations.The types of the detected blobs are manually annotated, at step 1306. Ifthe blob, at step 1307, is determined to be annotated as a triangle, therelative positioning of its left and right vertices, at step 1309, areexamined, to verify that it has the right orientation. If not, necessaryoperations, at 1311, are performed to rotate it back to its rightposition. For example, FIG. 11 shows that the triangular sign isslightly tilted to the right. Therefore, to correct this tilt, the blobis rotated around its right vertex and the upright version shown in FIG.12 is obtained.

A list of features, at step 1308, is extracted from each detected blob.Such feature vectors (plus the proper class labels) are then used astraining samples for SVM learning at step 1310. In one embodiment, theLibSVM tool (see, for example, P. Chen, C. Lin, and B. Scholkopf, “Atutorial on v-support vector machines”, Applied Stochastic Models inBusiness and Industry, 21:111-136, 2005. which is hereby incorporated byreference in its entirety), can be used with the kernel chosen to beRadial Basis Function (RBF).

During the classification phase, given a test video 1312, blob detectionand feature extraction, as discussed above, are performed, at steps 1314and 1316, respectively, to form testing feature vectors, at step 1318.Then the SVM classifier 906 such as the LibSVM tool, at step 2420,recognizes each blob type using the pre-trained SVM model.

With respect to feature extraction, a total of forty features areextracted from each blob, which aim to capture both of its geometric andsymmetric characteristics. For example, observing that the targettraffic signs all have regular shapes, and are horizontally symmetric,the first twenty features are used to capture its left-side shape, anduse the second twenty features to reveal its symmetry. For simplicity,the four vertices 1402, 1404, 1406, 1408 of a blob's bounding box aredenoted A, B, C and D as shown in FIG. 14. The first twenty features areextracted as the twenty equally sampled distances from its left edge(i.e. AB) to the blob. FIG. 14 illustrates the first three distances|AE|, |A₁E₁| and |A₂E₂|.

The same process is repeated by measuring the distances from the rightedge (i.e. CD) to the blob, and obtain another set of twenty distances.The absolute difference is then taken between every two distances thatsit on the same horizontal line, but represent the distances from theleft and right, respectively. In example of FIG. 14, the first threedifferential distances are ∥AE|−|DE∥, ∥A₁E₁|−|D₁F₁∥, and ∥A₂E₂|−|D₂F₂∥.The set of such differential distances then form the second twentyfeatures.

FIG. 24 is an operational flow diagram illustrating one process fordetecting an object of interest, such as a traffic sign using theSVM-based detector 114. The operational flow diagram begins at step 2402and flows directly to step 2409. A frame/image 110, at step 2404, isreceived by the SVM-based detector 114. The red channel I^(R), at step2406, is extracted for the incoming frame/image 110. One or more blobs,at step 2408, are detected in the extracted color channel, as discussedabove. Moreover, observing that some traffic signs are failed to bedetected due to their broken edges in I^(R), another set of blobs aredetected from its morphologically dilated image. The two sets of blobsare then merged together and redundant blobs are removed.

Each blob in the merged set of blobs is then analyzed to determine ifthe rough shape (or approximate shape) of the blob is potentially anobject of circular, triangular, or inverted triangular shape. If not,the blob is removed from further processing. For example, this roughshape analysis is performed by examining if certain areas within theblob's bounding box only contain background. For instance, if the roughshape of the blob is a circular object, then the four small corners1502, 1504, 1506, 1508 areas as shown in FIG. 15 should not contain anypixels from the object. Areas 1510, 1512, 1514, 1516 are used forchecking triangular objects. This process is advantageous because itdistinguishes triangular objects from others, speeds up the process, andhelps reduce false alarms.

When a blob is determined to possibly be of a triangular shape, theorientation process discussed above is performed. Feature extraction, atstep 2410, is performed to form a test sample. The pre-trained SVMmulti-classifier (to detect circle, triangle, and inverted triangle), atstep 2412, is then applied to each test sample. As discussed above,individual pre-trained classifiers can also be applied instead of amulti-class classifier to detect each target object type. The class thathas the highest probability is then returned, at step 2414. TheSVM-based detector, at step 2416, visually indicates the location of thedetected object (e.g., traffic sign) corresponding to the blob in theframe/image 110. For example, a bounding box similar to that shown inFIG. 11 can be displayed around the sign 2102 in FIG. 21. The controlflow then exits at step 2418.

In one or more embodiments, a tracking approach is applied to maintain aconsistent detection and improve performance. For example, for any blobB in the current frame, if the blob's classification probability fromthe multi-classifier (or individual classifiers) is lower than a giventhreshold such as 0.5 (i.e. if a confident classification is not able tobe made) an attempt to match this blob with blobs in the preceding frameis made. If a good match is found (e.g., B^(p)), meaning that theycorrespond to the same traffic sign, B is assigned to the same class asB^(p). The object tracking and partial scanning processes discussedabove with respect to the feature-based detector 112 are also applicableto the SVM-based detector 114 as well.

In addition to the SVM-based detector 114 discussed above, the ODS 104can also comprise a boosted-cascade detector 116. In this embodiment,the boosted-cascade detector 116 utilizes an Adaptive Boost (AdaBoost)based framework that is extended to include both (or at least one of)Haar and HOG (Histograms of Oriented Gradients) features. For example,as will be discussed in greater detail below, a cascade of classifiersare trained using both Haar features and HOG based weak classifiers in adiscriminative fashion. Also, the feature selection process is decoupledfrom the ensemble classifier design to form an ensemble classifier. Thispermits scalable training of strong classifiers to require significantlyless compute power and also achieves desired performance with a smallernumber of features.

The boosted-cascade detector 116 utilizes the AdaBoost-based frameworkto classify an image patch into either an “object” or a “non-object”.More specifically, the AdaBoost-based framework is applied to select(and weight) a set of weak classifiers, some of which, may be based onHaar wavelet features. These features consider adjacent rectangularregions at a particular location in a window and sum up the pixelintensities in these regions. The differences between these regions arethen calculated in order to categorize subsections of a frame/image

These wavelet features can be applied to patches of gray-scale images,and parameterized by their geometric properties such as position, widthand height. In one embodiment, five Haar wavelets are used. Thesewavelets are two edge detectors (rectangle pairs), horizontal andvertical; two bar detectors (rectangle triples), horizontal andvertical, and one center-surround detector (rectangle within rectangle).FIG. 16 shows the above Haar features. For example, FIG. 16 shows thehorizontal and vertical edge detector features 1602, 1604, thehorizontal and vertical bar detector features 1606, 1608, and the centersurround feature 1610. Pixels below the white areas are weighted by +1,and −1 for the black areas. A more detailed discussion on the AdaBoostframework is given in Viola, P., Jones, M.: Robust real-time objectdetection: Technical Report CRL 2001/01, Cambridge Research Laboratory(2001), which is hereby incorporated by reference in its entirety.

In addition to Haar features, the weak classifiers are also trained onHOG features with respect to the horizontal, vertical, and diagonaldirections. HOG features can be used to count occurrences of gradientorientation in localized portions of a frame/image. FIG. 17 shows theHOG features discussed above. FIG. 17 shows how to calculate ahorizontal Hog feature 1702, a first diagonal HOG feature 1704, avertical HOG feature 1706, and a second diagonal HOG feature 1708. Forexample, the first HOG 1702 shown is designed to capture the gradientstrength along the approximate horizontal direction. A more detaileddiscussion on HOG is given in Dalal, N., Triggs, B.: Histograms oforiented gradients for human detection: CVPR (2005), which is herebyincorporated by reference in its entirety.

In FIGS. 16 and 17, each outer rectangle 1612, 17010 indicates an imagepatch and the sub-region 1614, 1712 (w×h) within it indicates theposition for feature computation. Note that during AdaBoost training,for each image patch, all possible sizes and positions of suchsub-regions are enumerated and the optimal one is learned. During thetraining process, a cascade classifier is trained which contains acascade of rejectors. At each layer of this classifier, thecascade-boosted based detector 116 uses an AdaBoost process to train anensemble classifier based on a set of weak classifiers. At the end ofthe training process, as similar to the SVM-based detector 114 discussedabove, one or more cascade classifiers will be trained for objects ofinterest. For example, if traffic signs are objects of interest, aclassifier for circles, a classifier for triangles, and a classifier forinverted triangles will be trained. However, it should be noted that forother applications, other classifiers will be trained, such asclassifiers for detecting vehicles, pedestrians, animals, headlights,and/or the like.

During the classification (recognition/localization) process, given animage frame which may be in color, the frame is first converted tomonochrome (gray-scale) and subsequently scanned with a sliding windowof different sizes at different locations. The image patch within eachwindow is then extracted and classified. More specifically, thecascade-boosted based detector 116 starts the window size at an initialsize (e.g., 24×24); then for each subsequent round of scanning, the sizeis increased by a constant percentage (e.g. 25%). Nevertheless, eachwindowed patch is ultimately scaled to the initial size (24×24), andvariance-normalized before the feature calculation, to minimize theeffect of illumination change. As it is very likely that a sign isdetected in multiple overlapping windows at various scales, a fusionmechanism is finally applied to localize it with a single size andplace.

FIG. 18 shows one example of the cascade-boosted detector 116. Inparticular, FIG. 18 shows that the cascade-boosted detector 116comprises a multi-level detection framework. Cascaded classificationmodule 1802 receives an incoming frame/image 110. 1802 includes a set ofclassifiers 1804 that have been trained on Haar and HOG features toidentify objects of interest within the incoming frame/image 110. Forexample, the cascaded classification module 1802 utilizes asliding-window search scheme where windows of various scales are slidacross the frame/image 110. The rectangular image patch underneath thesliding window is checked by one or more weak classifiers from the setof classifiers 1804 as the sliding window moves across the frame/image110 to distinguish target object from non-target object patches based onthe Haar and/or HOG feature(s) for which the classifier has beentrained.

For example, the sliding window is a rectangle of a certain aspect ratio(for instance, 1:1 or 4:3). Wherever the window is placed and whateversize it is, the window can be rescaled (normalized) to a constant size(such as 50×50, or 160×120). Relative to this standard size, each Haarand/or HOG weak classifier in the set of classifiers 1804 references arectangle within it. Each weak classifier specifies that some particularkind of calculation is made on the values of the pixels in thatrectangle, and the result is a single number (a “scalar”). The type ofcalculation depends of the type of Haar or HOG feature being searchedfor, as has already been explained in the discussions of FIGS. 16 and17. The weak classifier also specifies a threshold, and an inequalitysense (which is either “less than or equal to” or “greater than”). Theresult of the calculation is compared to the threshold, and checked tosee if the specified sense holds. If the sense holds, this is evidencethat the sliding window contains an instance of the target/desiredobject class. If the sense does not hold, this is evidence that thesliding window does not contain an instance of the target/desired objectclass (e.g., a circle class, a triangle class, and an inverted triangleclass, in the case of sign detection). A real instance of the objectclass usually triggers multiple detections around it, which overlap witheach other; these are merged into a single detection in a later layer.There may also be some false detections, which are usually separate fromone another, and can be removed at higher level of decision making.

In applications in which at most only one instance of the desired objectclass can be present in the image frame, or at most only one instance(the most prominent, or the most important, for instance), frame-levelobject detection layer 1806 of the cascade-boosted detector 116 can beincluded, which can then present a detected object of interest to a userby calculating a bounding box 1902 for the detected object 1904, asshown in FIG. 19. (Traffic signs are used in this figure only asexamples, for the sake of explanation.)

FIG. 20 illustrates a flowchart for the above detection processperformed by the detector 116. Incoming frame/image 110 is received bythe cascaded classification module 1802. A sliding window, at step 2004,is iteratively applied over the frame/image 110. For each sliding windowposition and size the window, at step 2006, is tested by applying one ormore weak classifiers from the set of weak classifiers 1804 that havebeen trained on Haar and HOG features, according to the cascade layeringarchitecture which will now be discussed.

For each layer of the cascade classifier (module), the following isperformed. All weak classifiers of that layer, at step 2008, are appliedto the window. (There are some AdaBoost architectures where onlyselected weak classifiers are applied in a decision tree-like manner.)The decisions from all of the weak classifiers as to whether a targetobject has been detected or not detected are unified into a singledecision, at step 2010. In step 2012, if a target object has not beendetected, process 2006 exits, but if a target object has been detectedby all layers up to and including the current one, the processdetermines if there is another layer. If there is another layer, theprocess returns to step 2008. If there is not another layer, the slidingwindow position is declared finally to be an instance of the desiredobject class, and is added to the result set. A non-maximal suppressionoperation, at step 2014, is performed to reduce the result set size byeliminating highly-overlapping window positions. The remaining set ofzero or more detected target objects 2016 is output.

As discussed above, the set of weak classifiers 1804 used to buildcascaded classifier 1802 have been trained, based on Haar and HOGfeatures. The following explains how this training is performed. Theperformance (measured by detection rate (DR) and false acceptance rate(FAR)) of the detector 116 in FIG. 1 (also referred to as the Cascadedclassifier 1802 in FIG. 18) depends on the performance of the set ofselected weak classifiers 1804. In addition, the number of sub-imagesneeded to be evaluated per frame against a classifier is very large(approximately a million sub-images for a frame size of 640×480).Therefore, a classifier needs to be run over a frame at extremely highspeed (e.g., 24 fps) for a real-time application. Therefore, one or moreembodiments train a cascade of classifier layers, each layer composed ofsteps 2008/2010/2012 in FIG. 20, of various complexity.

To train an AdaBoost classifier with a desired detection rate (DR) andfalse accept rate (FAR), a positive sample set (target object imagepatches) and a negative sample set (non-target object image patches) areused. To achieve a desired performance at high detection speed, oneembodiment trains a series of classifiers, each of which is trained tohave very high DR and moderate FAR. For example, in one embodiment, eachcascade is trained to achieve a FAR of approximately 50% in each layer,with n total layers, where n=log 2(F); F is the desired false positiverate: 1×10⁻⁶. A sub-image that is rejected at a certain layer of thecascade is rejected forever without the need for further tests, as shownin FIG. 21. In this way, a good percentage of non-vehicle sub-images arerejected in the first few layers where relatively simple classifiers areevaluated. Consider the fact that most of the sub-images are negatives.This strategy saves a lot of time.

The positive training samples for each layer can be the same, orsubstantially the same, for all layers, and are supplied by annotatedsamples. Negative samples are produced automatically, on the fly foreach classifier layer, by randomly cropping from annotated videos, butavoiding overlapping with regions annotated as target objects by, forexample, 40% or more. A qualified negative sample for a classifier fortraining layer must have been accepted by all previously trainedclassifier layers. To extract a negative sample set from the collectionfor training the first layer, sub-images of different sizes are selectedfrom randomly-chosen frames from the video collection, where thesample's position (as defined, for example, by its center) within theframe is also randomly chosen, subject to sample size (the sample has tolie completely within the video frame), and its size is chosen from arandom distribution which is shaped to choose more small sizes thanlarge, to correspond to the fact that real world objects (like trafficsigns, for example) are more likely to be distant from the camera thannear to it. (The exact shaping of the distribution will beapplication-dependent, as some applications are 2-D; some are 3-D butimpose constraints on object location, etc.)

The features used for learning are the five Haar and four HOG featuresdiscussed above with respect to FIGS. 16 and 17. The same rectangularregions generated for the Haar features are also used for the HOGfeatures; in other words, a common rectangle-generator is used for both.This rectangle-generator can, for instance, generate (propose)rectangles of dimensions 2×2, 2×3, 2×4, . . . , 3×2, 3×3, 3×4, . . . ,up to a maximum size, which is the size of the normalized window

Weak classifier rectangle placement: once the rectangle dimensions areset, the rectangle-generator first proposes placing the rectangle sothat it is aligned with, a given region, such as the upper left-handcorner, of the normalized window. Then, rectangle-generator proposesplacing the rectangle one position to the right, and then two positionsto the right, etc, until the rectangle no longer fits inside the rightend of the normalized window. The rectangle-generator then proposesplacing the rectangle at the left end of the second row in thenormalized image, and so on, until the bottom of the normalized windowis reached. In the end, all valid positions are proposed.

For each proposal of a rectangle size and rectangle placement, each ofthe HOG and Haar features are evaluated to produce the scalar value foreach of the training instances. The four HOG features are computedtogether, as follows, although they can be selected for useindividually. Over each rectangle generated by the above procedure, therelative strengths of intensity gradients of the four orientationshorizontal, vertical, and the two principle diagonals (See FIG. 17) arecomputed, and normalized so that the sum of the strengths is 1. For thevertically-oriented gradient, for instance, for every pair of pixels inthe rectangle that are vertically adjacent (one above the other,corresponding to 1706 in FIG. 17), the gray level (brightness) of thelower pixel is subtracted from the gray level of the upper pixel, andthese differences are summed over all such pixel pairs in the rectangle.The absolute value of the sum is taken. Similarly for the otherorientations. These are normalized by dividing all sums by the sum ofthe four sums. So if the horizontal, vertical, slash diagonal andbackslash diagonal sums were 10, 20, 0, and 10, they would normalize to0.25, 0.5, 0, and 0.25. So, the value of the first HOG feature would be0.25, of the second, 0.5, and so on. Although there are four HOGfeatures, only three of them are actually independent

Once each of the HOG and Haar features are evaluated, the optimalthreshold and the optimal inequality sense are calculated for eachtraining instance. The term “optimal” means that, ideally, for all thepositive training instances, their scalar value stands in the specifiedinequality relation to their threshold, and for all the negativetraining instances, the scalar value stands in the opposite relation tothe threshold. Typically, this ideal condition is not met, so theAdaBoost process, in one embodiment, selects, at each cycle of selectingthe best next weak classifier, the one feature that comes closest tobeing optimal. This feature can either be a Haar or HOG feature. Inother words, the one feature (for the current rectangle), is selectedthat is most informative about discriminating between the positive andnegative examples without regard to which kind of feature it is.

Weak classifiers are collected by this method until a number of themtogether give the correct decision over the training set a high enoughpercent of the time that the layer performance requirements have beenmet. In other words, the positive examples are labeled positivesufficiently often (the DR) and the negative examples are labeledpositive sufficiently infrequently (the FAR). When these two conditionsare met, the layer is complete.

Multiple layers are accumulated into a “cascade” by this process untilthe overall performance requirements are achieved. A cascade of N layershas a performance where the overall DR is the product of the detectionrates of the individual layers and the overall FAR is the product of thefalse acceptance rates of the individual layers. If all layers have thesame performance, the overall DR is the DR of one layer taken to the Nthpower, and similarly for the FAR. If the DR for each layer is nearly 1,and the FAR is close to zero, the addition of each new layer has theeffect of driving the FAR lower and lower, while leaving the DR onlyvery slightly diminished.

With respect to feature selection during the training process discussedabove, one embodiment utilizes a Forward Feature Selection (FFS)process. This allows fewer features to be used than conventionalAdaBoost systems while achieving the same or better performance. Oneexample of a FFS process is given in J. Wu, C. Brubaker, M Mullin, andJ. Rehg, Fast asymmetric learning for cascade face detection, IEEETrans: on Pattern Analysis and Machine Intelligence, 30(3), 369-382,January 2008, which is hereby incorporated by reference in its entirety.One embodiment of the present invention uses the FFS process to buildclassifier cascade layers. This separates apart two functions which areintermingled in classic AdaBoost trainers: feature selection and featureweighting.

Unlike AdaBoost, the FFS process does not change the trainingpopulation, and all selected features (weak classifiers) are treatedequally in the ensemble. Therefore, by implementing the FFS processduring AdaBoost training, all the features can be trained just once, andthe results saved in a table in which a row corresponds to a feature anda column corresponds to a sample. Features can be selectedincrementally, by inspecting the table. The candidate features arecompared to find the one that, by adding it to the ensemble (which isinitially empty), produces the lowest classification error. Since thenumber of votes of the ensemble for any sample is an integer in therange [0, t], where t is the number of features in the ensemble,histograms can be calculated to determine the threshold for theensemble, instead of having to perform a sort.

A feature weighting process is also used in one embodiment. One exampleof a feature weighting process is Linear Asymmetric Classifier (LAC),which is further discussed in Wu et al. In one embodiment, each layer isan ensemble classifier where all chosen weak classifiers (features) areevaluated, and a weighted sum of the results is compared against thethreshold of the layer classifier to determine acceptance or rejection.The coefficients are chosen to maximize the detection rate, whileholding the false accept rate at, for example, 50%. The false acceptrate of the entire classifier can be driven arbitrarily low by use ofmultiple layers. This treatment of the coefficients assigns far lesscost to a false accept than to a false reject, which is advantageous tohave when building a sliding window localizer (which typically generatesvastly more windows which are negative samples than positive). The fewpositive examples are important and should not be lost.

In some instances, as features are added to the ensemble, the detectionrate can oscillate and sometimes not converge. This can be traced to thefact that sometimes FFS selects two features that produced the sameclassification over the positive training set. An offsetting λ can beadded to the diagonal matrix elements to overcome this, but even if afairly large λ is used, some weak classifier weights can turn negative.Even when two features produced are very similar, although notidentical, classifications, calculating the inverse matrix becomesnumerically unstable.

Therefore, one embodiment overcomes this numerical instability problemas follows. In the feature selection layer, for every new feature chosenby FFS, its classification is compared with the classifications ofpreviously selected features, one by one. If one difference is less thana threshold (e.g. 5%), the new feature is discarded. And in the LACformation layer, if it is found that a weak classifier has a negativeweight, it is removed from the ensemble classifier. Since these twomethods operate at different times in the training, both of them can beused.

As can be seen, the above training process results in a multiclassclassifier whose basic elements are simple ensemble classificationtrees. The root of such a tree is an ensemble (set) of all classes thatwere trained for (one of which can be a “none of the above” class, whichrepresents failure to find any object of interest). At each node in thetree, there is a test and binary decision, each branch of which denotesan ensemble which is a subset of its ancestor ensemble. Eventually thisgrounds out in single classes.

The test at each node is constructed (during training) as theapplication of a feature to the input and comparison of its numericalvalue to a threshold. The feature to be selected for the test is the onethat is most informative (gives the cleanest separation of classes,based on the training samples) that has not already been used. Inparticular, all possible Haar and HOG features have been evaluated overthe training data set, and are available for selection, they can befreely intermixed at successive nodes, as shown in FIG. 22. Inparticular, FIG. 22 shows an AdaBoost-trained classifier. The columnsare the layers in the cascade, the leftmost being Layer 1. The columnheading is the number of weak classifiers in that layer. The rowsrepresent the feature (weak classifier) selections within each layer, inorder. Large squares indicate that a HOG feature was selected; smallsquares indicate that a Haar feature was selected.

FIG. 23 is an operational flow diagram for the training process above.It should be noted that a more detailed discussion on each step of theflow diagram in FIG. 23 has already been given above. The operationalflow begins at step 2302 and flows directly to step 2304. The trainingprocess, at step 2304, receives positive and negative samples. Thetraining process, at step 2308, generates rectangles up to a maximumwidth and height that are defined by a normalized window size which therectangles are to be placed.

The training process, at step 2310, evaluates each Haar and HOG featurefor each proposal of a rectangle size and rectangle placement within thenormalized window(s). A scalar value, at step 2312, is produced as aresult of the evaluation. The training process, at step 2314, calculatesan optimal threshold and an optimal inequality sense. The trainingprocess, at step 2316, selects, at each cycle of selecting a weakclassifier, the feature that is closest to being optimal. The trainingprocess, at step 2318, collects weak classifiers for each level, so thatthe correct decision over the training set is obtained frequently enoughsuch that the “layer” performance requirements have been met. Thetraining process, at step 2320, accumulates multiple layers into a“cascade” classifier until the overall performance requirements areachieved. The control flow then exits at step 2322.

Information Processing System

FIG. 25 is a block diagram illustrating an information processing systemthat can be utilized in embodiments of the present invention. Theinformation processing system 2500 is based upon a suitably configuredprocessing system adapted to implement one or more embodiments of thepresent invention (e.g., the system 102 of FIG. 1). Any suitablyconfigured processing system can be used as the information processingsystem 2500 in embodiments of the present invention.

The information processing system 2500 includes a computer 2502. Thecomputer 2502 has a processor(s) 2504 that is connected to a main memory2506, mass storage interface 2508, and network adapter hardware 2510. Asystem bus 2512 interconnects these system components. Although only oneCPU 2504 is illustrated for computer 2502, computer systems withmultiple CPUs can be used equally effectively. The main memory 2506, inthis embodiment, comprises the object detection system 104 and itscomponents, the image database 108, and the images 110.

The mass storage interface 2508 is used to connect mass storage devices,such as mass storage device 2514, to the information processing system2500. One specific type of data storage device is an optical drive suchas a CD/DVD drive, which can be used to store data to and read data froma computer readable medium or storage product such as (but not limitedto) a CD/DVD 2516. Another type of data storage device is a data storagedevice configured to support, for example, NTFS type file systemoperations.

An operating system included in the main memory is a suitablemultitasking operating system such as any of the Linux, UNIX, Windows,and Windows Server based operating systems. Embodiments of the presentinvention are also able to use any other suitable operating system. Someembodiments of the present invention utilize architectures, such as anobject oriented framework mechanism, that allows instructions of thecomponents of operating system to be executed on any processor locatedwithin the information processing system 2500. The network adapterhardware 2510 is used to provide an interface to a network 2511.Embodiments of the present invention are able to be adapted to work withany data communications connections including present day analog and/ordigital techniques or via a future networking mechanism.

Although the exemplary embodiments of the present invention aredescribed in the context of a fully functional computer system, those ofordinary skill in the art will appreciate that various embodiments arecapable of being distributed as a program product via CD or DVD, CD-ROM,or other form of recordable media, or via any type of electronictransmission mechanism. Also, aspects of the present invention may takethe form of an entirely hardware embodiment, an entirely softwareembodiment (including firmware, resident software, micro-code, etc.), oran embodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit” “module” or “system”.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium A computer readable storagemedium may be, for example, but not limited to, an electronic, magnetic,optical, electromagnetic, infrared, or semiconductor system, apparatus,or device, or any suitable combination of the foregoing. More specificexamples (a non-exhaustive list) of the computer readable storage mediuminclude computer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), an optical fiber, a portable compact disc read-onlymemory (CD-ROM), an optical storage device, a magnetic storage device,or any suitable combination of the foregoing. A computer readablestorage medium may be any tangible medium that can contain, or store aprogram for use by or in connection with an instruction executionsystem, apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (e.g., through the Internet using an Internet ServiceProvider).

NON-LIMITING EXAMPLES

Although specific embodiments of the invention have been disclosed,those having ordinary skill in the art will understand that changes canbe made to the specific embodiments without departing from the spiritand scope of the invention. The scope of the invention is not to berestricted, therefore, to the specific embodiments, and it is intendedthat the appended claims cover any and all such applications,modifications, and embodiments within the scope of the presentinvention.

1. A method, with an information processing system, for a digital imagebased object detection system, the method comprising: receiving at leastone image representing at least one frame of a video sequence of anexternal environment; extracting a given color channel of the at leastone image; identifying at least one blob that stands out from abackground of the given color channel that has been extracted;extracting one or more features from the at least one blob that has beenidentified; providing the one or more features that have been extractedto a plurality of pre-learned object models each comprising a set ofpre-defined features associated with a pre-defined blob type; comparing,in response to the providing, the one or more features to the set ofpre-defined features of each pre-learned object model; determining,based on the comparing, that the at least one blob is of a type thatsubstantially matches a pre-defined blob type associated with one of thepre-learned object models; and visually indicating, in response to thedetermining, at least a location of an object within the at least oneimage that corresponds to the at least one blob.
 2. The method of claim1, wherein the object is a traffic sign.
 3. The method of claim 1,further comprising: morphologically dilating the at least one image;detecting at least a second blob from the at least one image that hasbeen morphologically dilated; if the at least one blob and the secondblob refer to an identical object, merging the at least one blob and thesecond blob into one blob; and if the at least one blob and the secondblob refer to different objects, maintaining the at least one blob andthe second blob as separate blobs.
 4. The method of claim 1, furthercomprising: applying a bounding box around the at least one blob;analyzing two or more corner regions of the bounding box; anddetermining a rough shape of the at least one blob based on the two ormore corner regions comprising one of a portion of the backgroundportion and a potential portion of the at least one blob.
 5. The methodof claim 4, wherein the at least one blob is determined to potentiallybe one of a circular shape, a triangular shape, and an invertedtriangular shape.
 6. The method of claim 1, further comprising:determining that a classification probability of the at least one blobwith respect to the plurality of pre-learned object models is below agiven threshold; comparing the at least one blob with one or more blobsfrom at least one preceding image that has been assigned a blob typefrom the plurality of pre-learned object models; determining that the atleast one blob substantially matches with at least one of the one ormore blobs; and assigning the at least one blob to the blob type thathas been assigned to the at least one of the one or more blobs in the atleast one preceding image.
 7. The method of claim 1, wherein the one ormore features are associated with a geometric and symmetriccharacteristics of the at least one blob.
 8. The method of claim 1,wherein extracting the one or more features comprises: sampling a givennumber of distances from a left edge of a bounding box surrounding theat least one blob to the at least one blob; sampling a given number ofdistances from a right edge of the bounding box to the at least oneblob; and calculating an absolute difference between every two distancesthat reside on a same horizontal line and represent a set of distancesfrom the left edge and the right edge of the bounding box.
 9. The methodof claim 1, further comprising: tracking the object for at least one ormore subsequent images, wherein the tracking comprises: identifying asearch area S in the one or more subsequent images; and localizing theobject within S.
 10. The method of claim 9, wherein localizing theobject within S comprises: substituting the search area S for the atleast one image in the receiving step.
 11. The method of claim 9,further comprising: dividing the at least one image into a plurality ofquadrants; searching the search area S and one of the quadrants; andidentifying a new object within the image in response to searching thesearch area S and one of the quadrants.
 12. The method of claim 9,wherein identifying the search area S comprises: determining a size of Sbased on a size of the object within the at least one image, and amoving direction of the object.
 13. The method of claim 12, wherein themoving direction is based on a displacement of the object in anx-direction and a y-direction in a preceding image of the at least oneimage.
 14. The method of claim 12, wherein the size of the search area Sis further determined based on a size change of the object from apreceding image to the at least one image and a size of the object inthe at least one image.
 15. An information processing system fordetecting objects in a digital image, the information processing systemcomprising: a memory; a processor communicatively coupled to the memory;and an object detection system communicatively coupled to the memory andthe processor, the object detection system configured to perform amethod comprising: receiving at least one image representing at leastone frame of a video sequence of an external environment; extracting agiven color channel of the at least one image; identifying at least oneblob that stands out from a background of the given color channel thathas been extracted; extracting one or more features from the at leastone blob that has been identified; providing the one or more featuresthat have been extracted to a plurality of pre-learned object modelseach comprising a set of pre-defined features associated with apre-defined blob type; comparing, in response to the providing, the oneor more features to the set of pre-defined features of each of theplurality of pre-learned object models; determining, based on thecomparing, that the at least one blob is of a type that substantiallymatches a pre-defined blob type associated with one of the pre-learnedobject models; and visually indicating, in response to the determining,at least a location of an object within the at least one image thatcorresponds to the at least one blob.
 16. The information processingsystem of claim 15, wherein the method performed by the informationprocessing system further comprises: applying a bounding box around theat least one blob; analyzing two or more corner regions of the boundingbox; and determining a rough shape of the at least one blob based on thetwo or more corner regions comprising one of a portion of the backgroundportion and a potential portion of the at least one blob.
 17. Theinformation processing system of claim 13, wherein the method performedby the information processing system further comprises: determining thata classification probability of the at least one blob with respect tothe plurality of pre-learned object models is below a given threshold;comparing the at least one blob with one or more blobs from at least onepreceding image that has been assigned a blob type from the plurality ofpre-learned object models; determining that the at least one blobsubstantially matches with at least one of the one or more blobs; andassigning the at least one blob to the blob type that has been assignedto the at least one of the one or more blobs in the at least onepreceding image.
 18. The information processing system of claim 15,wherein the method performed by the information processing systemfurther comprises: tracking the object for at least one or moresubsequent images, wherein the tracking comprises: identifying a searcharea S in the one or more subsequent images; and localizing the objectwithin S.
 19. The information processing system of claim 18, whereinidentifying the search area S comprises: determining a size of S basedon a size of the object within the at least one image, and a movingdirection of the object, wherein the moving direction is based on adisplacement of the object in an x-direction and a y-direction in apreceding image of the at least one image, wherein the size of thesearch area S is further determined based on a size change of the objectfrom a preceding image to the at least one image and a size of theobject in the at least one image.
 20. A computer program product fordetecting objects in a digital image, the computer program productcomprising: a storage medium readable by a processing circuit andstoring instructions for execution by the processing circuit forperforming a method comprising: receiving at least one imagerepresenting at least one frame of a video sequence of an externalenvironment; extracting a given color channel of the at least one image;identifying at least one blob that stands out from a background of thegiven color channel that has been extracted; extracting one or morefeatures from the at least one blob that has been identified; providingthe one or more features that have been extracted to a plurality ofpre-learned object models each comprising a set of pre-defined featuresassociated with a pre-defined blob type; comparing, in response to theproviding, the one or more features to the set of pre-defined featuresof each pre-learned object model; determining, based on the comparing,that the at least one blob is of a type that substantially matches apre-defined blob type associated with one of the pre-learned objectmodels; and visually indicating, in response to the determining, atleast a location of an object within the at least one image thatcorresponds to the at least one blob.
 21. The computer program productof claim 20, wherein the instructions are further for: applying abounding box around the at least one blob; analyzing two or more cornerregions of the bounding box; and determining a rough shape of the atleast one blob based on the two or more corner regions comprising one ofa portion of the background portion and a potential portion of the atleast one blob.
 22. The computer program product of claim 20, whereinthe instructions are further for: determining that a classificationprobability of the at least one blob with respect to the plurality ofpre-learned object models is below a given threshold; comparing the atleast one blob with one or more blobs from at least one preceding imagethat has been assigned a blob type from the plurality of pre-learnedobject models; determining that the at least one blob substantiallymatches with at least one of the one or more blobs; and assigning the atleast one blob to the blob type that has been assigned to the at leastone of the one or more blobs in the at least one preceding image. 23.The computer program product of claim 20, wherein the instructions arefurther for: tracking the object for at least one or more subsequentimages, wherein the tracking comprises: identifying a search area S inthe one or more subsequent images; and localizing the object within S.24. The computer program product of claim 23, wherein identifying thesearch area S comprises: determining a size of S based on a size of theobject within the at least one image, and a moving direction of theobject, wherein the moving direction is based on a displacement of theobject in an x-direction and a y-direction in a preceding image of theat least one image, wherein the size of the search area S is furtherdetermined based on a size change of the object from a preceding imageto the at least one image and a size of the object in the at least oneimage.